SAS macros for point and interval estimation of area under the receiver operating characteristic curve for non-proportional and proportional hazards Weibull models

Aims and objectives  For prediction of risk of cardiovascular end points using survival models the proportional hazards assumption is often not met. Thus, non-proportional hazards models are more appropriate for developing risk prediction equations in such situations. However, computer program for evaluating the prediction performance of such models has been rarely addressed. We therefore developed SAS macro programs for evaluating the discriminative ability of a non-proportional hazards Weibull model developed by Anderson (1991) and that of a proportional hazards Weibull model using the area under receiver operating characteristic (ROC) curve.

Method  Two SAS macro programs for non-proportional hazards Weibull model using Proc NLIN and Proc NLP respectively and model validation using area under ROC curve (with its confidence limits) were written with SAS IML language. A similar SAS macro for proportional hazards Weibull model was also written.

Results  The computer program was applied to data on coronary heart disease incidence for a Framingham population cohort. The five risk factors considered were current smoking, age, blood pressure, cholesterol and obesity. The predictive ability of the non-proportional hazard Weibull model was slightly higher than that of its proportional hazard counterpart. An advantage of SAS Proc NLP in terms of the example provided here is that it provides significance level for the parameter estimates whereas Proc NLIN does not.

Conclusion  The program is very useful for evaluating the predictive performance of non-proportional and proportional hazards Weibull models.

Coronary artery stenoses: detection with calcium scoring, CT angiography, and both methods combined

PURPOSE: To investigate prospectively the relative accuracy of computed tomographic (CT) angiography, calcium scoring (CS), and both methods combined in demonstrating coronary artery stenoses by using conventional angiography as the reference standard. MATERIALS AND METHODS: The study was approved by the institutional review board Human Research Ethics Committee, and all patients completed written informed consent. Fifty patients (40 men, 10 women) aged 62 years ± 11 (± standard deviation) who were suspected of having coronary artery disease underwent both conventional coronary angiography and multisection coronary CT angiography with CS. Sensitivity and specificity of CS, CT angiography, and both methods combined in demonstrating luminal stenosis greater than or equal to 50% were determined for each arterial segment, coronary vessel, and patient. Receiver operating characteristic (ROC) curves were generated for CS prediction of significant stenosis, and the Mann-Whitney U test was used for comparison of CS between groups. RESULTS: When used with segment-specific electrocardiographic phase reconstructions, CT angiography demonstrated stenosed segments with 79% sensitivity and 95% specificity. Mean calcium score was greater in segments, vessels, and patients with stenoses than in segments, vessels, and patients without stenoses (P < .001 for all); nine (16%) of 56 stenosed segments, however, had a calcium score of 0. The patient calcium score correlated strongly with the number of stenosed arteries (Spearman {rho} = 0.75, P < .001). CS was more accurate in demonstrating stenosis in patients than in segments (areas under ROC curve were 0.88 and 0.74, respectively). CT angiography, however, was more accurate than CS in demonstrating stenosis in patients, vessels, and segments. The sensitivity and specificity of CS varied according to the threshold used, but when the calcium score cutoff (ie, >150) matched the specificity of CT angiography (95%), the sensitivity of CS in demonstrating stenosed segments was 29% (compared with 79% for CT angiography). Combining CT angiography with CS (at threshold of 400) improved the sensitivity of CT angiography (from 93% to 100%) in demonstrating significant coronary disease in patients, without a loss of specificity (85%); this finding, however, was not statistically significant. CONCLUSION: CT angiography is more accurate than CS in demonstrating coronary stenoses. A patient calcium score of greater than or equal to 400, however, can be used to potentially identify patients with significant coronary stenoses not detected at CT angiography.

Subjective well-being as an indicator for clinical depression

The Theory of Homeostasis posits that Subjective Well-being (SWB) is regulated by a dynamic biological mechanism, assisting to maintain a positive view of life. Further, the theory suggests that clinical depression is the loss of SWB due to the defeat of this homeostatic defence system. To test this hypothesis it was predicted that people who were diagnosed as clinically depressed with the Semi-structured Clinical Interview (SCID-1/NP) based on the DSM-IV-TR Axis 1 would have a Personal Well-being Index-Adult (PWI-A) score below the normative range (70–80% of scale maximum). Following ethical approval a sample of 146 men was obtained and each was assessed on the SCID-1/NP and on the PWI-A. Subjects diagnosed as having one of several pathologies such as post traumatic stress disorder, panic disorder, social phobia and specific phobia were found to score significantly lower on the PWI-A compared to participants who received no diagnosis. However, as the data did not discriminate between currently depressed and persons with other non-depressive psychopathologies, a Receiver Operating Characteristics (ROC) curve analysis was used to explore this data further. Results indicated that the PWI-A was significantly better than guessing in discriminating clinically depressed cases, but only just so. Therefore, while this research found support for the proposition that the loss of SWB indicated clinical depression, the PWI-A is not sufficiently specific for diagnosis, nor can it be concluded that all instances of depression is the failure of SWB.

Cardiovascular risk prediction model development and assessment

The aim of this article is to review the development and assessment of cardiovascular risk prediction models and to discuss the predictive value of a risk factor as well as to introduce new assessment methods to evaluate a risk prediction model. Many cardiovascular risk prediction models have been developed during the past three decades. However, there has not been consistent agreement regarding how to appropriately assess a risk prediction model, especially when new markers are added to an existing model. The area under the receiver operating characteristic (ROC) curve has traditionally been used to assess the discriminatory ability of a risk prediction model. However, recent studies suggest that this method has its limitations and cannot be the sole approach to evaluate the usefulness of a new marker. New assessment methods are being developed to appropriately assess a risk prediction model and they will be gradually used in clinical and epidemiological studies.

Raising the profile of depression in the workplace

The prevalence of depression in the Australian workforce is unknown. Epidemiological surveys (e.g., the National Health Survey and National Survey of Mental Health and Wellbeing) do not routinely include a depression scale and within the mental health field, few studies focus on depression and employment groups specifically. Although the inclusion of a direct measure of depression in  national surveys is preferable, the prevalence of depression may be inferred from short screening scales of general mental health. In this paper, scores on the  General Health Questionnaire (GHQ-12) and the Kessler psychological distress scale (K10) for a sample of employed persons were mapped onto the CES-D (Iowa) measure of depression. The results of this study indicate that the  recommended GHQ-12 cut-off point is appropriate for estimating work-related depression prevalence. However, the cut-off point on the K10 (the short-scale  currently used in Australian national surveys) may need to be substantially  reduced if scores on the K10 are to be used to identify workers at risk of  depression. The routine inclusion of a direct depression measure in national  surveys is recommended, particularly considering the number of employed persons in Australia and large proportion of the sample classified as depressed in this study.

Quality of life assessment in the community-dwelling elderly : validation of the Assessment of Quality of Life (AQoL) instrument and comparison with the SF-36

Measurement of Health-Related Quality of Life (HRQoL) of the elderly requires instruments with demonstrated sensitivity, reliability, and validity, particularly with the increasing proportion of older people entering the health care system. This article reports the psychometric properties of the 12-item Assessment of Quality of Life (AQoL) instrument in chronically ill community-dwelling elderly people with an 18-month follow-up. Comparator instruments included the SF-36 and the OARS. Construct validity of the AQoL was strong when examined via factor analysis and convergent and divergent validity against other scales. Receiver Operator Characteristic (ROC) curve analyses and relative efficiency estimates indicated the AQoL is sensitive, responsive, and had the strongest predicative validity for nursing home entry. It was also sensitive to economic prediction over the follow-up. Given these robust psychometric properties and the brevity of the scale, AQoL appears to be a suitable instrument for epidemiologic studies where HRQoL and utility data are required from elderly populations.

Waist circumference has heterogeneous impact on development of diabetes in different populations : Longitudinal comparative study between Australia and Iran

Aims : Comparing waist circumference (WC) role in diabetes risk prediction and diagnosis of metabolic syndrome (MS) in different populations.

Methods : Population-based samples from Australia (n = 9026) and Iran (n = 8259) were studied in 2000 and followed for 4 years. Follow-up attendance was 58% and mean age was 51 vs. 47. Pearson correlations calculated between WC and other MS components. ROC for the role of WC in the prediction of incident diabetes was used.

Results : Prevalences of MS (48% vs. 28%), an increased WC (58.5% vs. 54.5%), low HDL-C (35% vs. 11.2%), high triglyceride (52.2% vs. 29.6%) were significantly higher in Iran. Fasting glucose ≥5.6 mmol/L was higher in Australia (26% vs. 23%). Hypertension was no different (38%). Pearson correlations between WC and other MS components were stronger in Australians: FPG (0.32 vs. 0.2), HDL (0.47 vs. 0.16), TG (0.38 vs. 0.30) and SBP (0.38 vs. 0.36). Among women, area under ROC curve for WC as a predictor for diabetes was significantly higher for Australians (0.76 vs. 0.68, p < 0.001) with no difference among men (0.69 vs. 0.71, p = 0.4).

Conclusion : WC was more strongly related to other components of MS in Australia. Association between WC and MS or incident diabetes varies between ethnicities.

Sample subset optimization for classifying imbalanced biological data

Data in many biological problems are often compounded by imbalanced class distribution. That is, the positive examples may largely outnumbered by the negative examples. Many classification algorithms such as support vector machine (SVM) are sensitive to data with imbalanced class distribution, and result in a suboptimal classification. It is desirable to compensate the imbalance effect in model training for more accurate classification. In this study, we propose a sample subset optimization technique for classifying biological data with moderate and extremely high imbalanced class distributions. By using this optimization technique with an ensemble of SVMs, we build multiple roughly balanced SVM base classifiers, each trained on an optimized sample subset. The experimental results demonstrate that the ensemble of SVMs created by our sample subset optimization technique can achieve higher area under the ROC curve (AUC) value than popular sampling approaches such as random over-/under-sampling; SMOTE sampling, and those in widely used ensemble approaches such as bagging and boosting.

A calibration protocol for population-specific accelerometer cut-points in children

Purpose

To test a field-based protocol using intermittent activities representative of children's physical activity behaviours, to generate behaviourally valid, population-specific accelerometer cut-points for sedentary behaviour, moderate, and vigorous physical activity.
Methods

Twenty-eight children (46% boys) aged 10–11 years wore a hip-mounted uniaxial GT1M ActiGraph and engaged in 6 activities representative of children's play. A validated direct observation protocol was used as the criterion measure of physical activity. Receiver Operating Characteristics (ROC) curve analyses were conducted with four semi-structured activities to determine the accelerometer cut-points. To examine classification differences, cut-points were cross-validated with free-play and DVD viewing activities.
Results

Cut-points of ≤372, >2160 and >4806 counts•min−1 representing sedentary, moderate and vigorous intensity thresholds, respectively, provided the optimal balance between the related needs for sensitivity (accurately detecting activity) and specificity (limiting misclassification of the activity). Cross-validation data demonstrated that these values yielded the best overall kappa scores (0.97; 0.71; 0.62), and a high classification agreement (98.6%; 89.0%; 87.2%), respectively. Specificity values of 96–97% showed that the developed cut-points accurately detected physical activity, and sensitivity values (89–99%) indicated that minutes of activity were seldom incorrectly classified as inactivity.
Conclusion

The development of an inexpensive and replicable field-based protocol to generate behaviourally valid and population-specific accelerometer cut-points may improve the classification of physical activity levels in children, which could enhance subsequent intervention and observational studies.

Are we predicting the actual or apparent distribution of temperate marine fishes?

Planning for resilience is the focus of many marine conservation programs and initiatives. These efforts aim to inform conservation strategies for marine regions to ensure they have inbuilt capacity to retain biological diversity and ecological function in the face of global environmental change – particularly changes in climate and resource exploitation. In the absence of direct biological and ecological information for many marine species, scientists are increasingly using spatially-explicit, predictive-modeling approaches. Through the improved access to multibeam sonar and underwater video technology these models provide spatial predictions of the most suitable regions for an organism at resolutions previously not possible. However, sensible-looking, well-performing models can provide very different predictions of distribution depending on which occurrence dataset is used. To examine this, we construct species distribution models for nine temperate marine sedentary fishes for a 25.7 km2 study region off the coast of southeastern Australia. We use generalized linear model (GLM), generalized additive model (GAM) and maximum entropy (MAXENT) to build models based on co-located occurrence datasets derived from two underwater video methods (i.e. baited and towed video) and fine-scale multibeam sonar based seafloor habitat variables. Overall, this study found that the choice of modeling approach did not considerably influence the prediction of distributions based on the same occurrence dataset. However, greater dissimilarity between model predictions was observed across the nine fish taxa when the two occurrence datasets were compared (relative to models based on the same dataset). Based on these results it is difficult to draw any general trends in regards to which video method provides more reliable occurrence datasets. Nonetheless, we suggest predictions reflecting the species apparent distribution (i.e. a combination of species distribution and the probability of detecting it). Consequently, we also encourage researchers and marine managers to carefully interpret model predictions.

Testing the validity, reliability and utility of the Self-Administration of Medication (SAM) tool in patients undergoing rehabilitation.

Determination of patients' ability to self-administer medications in the hospital has largely been determined using the subjective judgment of health professionals.

Validation and development of a clinical prediction rule in clinically suspected community-acquired pneumonia

To develop a mathematical model to predict the probability of having community-acquired pneumonia and to evaluate an already developed prediction rule that has not been validated in a clinical scenario.

Support vector machine (SVM) based multiclass prediction with basic statistical analysis of plasminogen activators

Plasminogen (Pg), the precursor of the proteolytic and fibrinolytic enzyme of blood, is converted to the active enzyme plasmin (Pm) by different plasminogen activators (tissue plasminogen activators and urokinase), including the bacterial activators streptokinase and staphylokinase, which activate Pg to Pm and thus are used clinically for thrombolysis. The identification of Pg-activators is therefore an important step in understanding their functional mechanism and derives new therapies.

Characterization and prediction of issue-related risks in software projects

Identifying risks relevant to a software project and planning measures to deal with them are critical to the success of the project. Current practices in risk assessment mostly rely on high-level, generic guidance or the subjective judgements of experts. In this paper, we propose a novel approach to risk assessment using historical data associated with a software project. Specifically, our approach identifies patterns of past events that caused project delays, and uses this knowledge to identify risks in the current state of the project. A set of risk factors characterizing “risky” software tasks (in the form of issues) were extracted from five open source projects: Apache, Duraspace, JBoss, Moodle, and Spring. In addition, we performed feature selection using a sparse logistic regression model to select risk factors with good discriminative power. Based on these risk factors, we built predictive models to predict if an issue will cause a project delay. Our predictive models are able to predict both the risk impact (i.e. the extend of the delay) and the likelihood of a risk occurring. The evaluation results demonstrate the effectiveness of our predictive models, achieving on average 48%-81% precision, 23%-90% recall, 29%-71% F-measure, and 70%-92% Area Under the ROC Curve. Our predictive models also have low error rates: 0.39-0.75 for Macro-averaged Mean Cost-Error and 0.7-1.2 for Macro-averaged Mean Absolute Error.

Hierarchical gene selection and genetic fuzzy system for cancer microarray data classification

This paper introduces a novel approach to gene selection based on a substantial modification of analytic hierarchy process (AHP). The modified AHP systematically integrates outcomes of individual filter methods to select the most informative genes for microarray classification. Five individual ranking methods including t-test, entropy, receiver operating characteristic (ROC) curve, Wilcoxon and signal to noise ratio are employed to rank genes. These ranked genes are then considered as inputs for the modified AHP. Additionally, a method that uses fuzzy standard additive model (FSAM) for cancer classification based on genes selected by AHP is also proposed in this paper. Traditional FSAM learning is a hybrid process comprising unsupervised structure learning and supervised parameter tuning. Genetic algorithm (GA) is incorporated in-between unsupervised and supervised training to optimize the number of fuzzy rules. The integration of GA enables FSAM to deal with the high-dimensional-low-sample nature of microarray data and thus enhance the efficiency of the classification. Experiments are carried out on numerous microarray datasets. Results demonstrate the performance dominance of the AHP-based gene selection against the single ranking methods. Furthermore, the combination of AHP-FSAM shows a great accuracy in microarray data classification compared to various competing classifiers. The proposed approach therefore is useful for medical practitioners and clinicians as a decision support system that can be implemented in the real medical practice.